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Abstract 

Concept map (CM) is a theoretically sound yet easy to learn tool and can be effectively used to represent 
knowledge. Even though many disciplines have adopted CM as a teaching and learning tool to improve 
learning effectiveness, its application in IS curriculum is sparse. Meaningful learning happens when one 
iteratively integrates new concepts and propositions into her existing cognitive structure. It is the 
process of how one acquires deep and applicative knowledge in certain domains such as Information 
Systems (IS). As important as meaningful learning is in IS education, there is a scarcity of method to 
assess it effectively. This study reports a series of experiments of adopting CM as a tool to enhance and 
evaluate students' learning, especially meaningful learning in IS education. Based on theoretical 
foundation of CMs and prior related empirical work, we designed a series of assignments that require 
students to complete CMs in three participating courses. We also designed and implemented a tool to 
help analyzing the CMs with certain level of automation. The completed CMs are collected and analyzed 
to answer our research questions. We believe the results demonstrate the utility of CMs in IS education 
as an effective tool to understand and assess students' meaningful learning. Our work also experimented 
with various methods to use CMs and the findings provide valuable insights as to how CM-based teaching 
and learning tools can be integrated into IS curricula seamlessly. 

Keywords: Concept map, meaningful learning, assessment, information systems education, 
pedagogical tool. 


1. INTRODUCTION 

In the ACM & AIS Curriculum Guidelines (Topi et 
al., 2010) for Undergraduate Degree Programs in 
Information Systems (IS), critical thinking (CT) is 
listed as one of the five foundational knowledge 
and skills. CT skills must be acquired through 
meaningful learning (Mayer, 2002), during which 
students acquire and build knowledge and 


cognitive processes, which are needed for them 
to become effective problem solvers in IS fields. 
Therefore, it is essential for IS educators to 
understand the nature and assess the quality of 
meaningful learning in order to design teaching 
artifacts that foster effective problem solving 
skills. 
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Meaningful learning was identified by Ausubel 
(Ausubel, 1963) as the most important learning 
principle. It is signified by integrating new 
concepts and propositions with existing relevant 
ideas in some substantive ways, within one's 
cognitive structure. This is an iterative process in 
which learners must continue to refine, rectify, 
rearrange, and reorganize the content and 
structure of their knowledge so that their 
cognitive structure can be improved. Opposite to 
rote learning (Novak, 1993; Novak & Gowin, 
1984), meaningful learning can be signified by: 
(1) Includes clarification of relations between 
concepts; (2) Involves self-assisted learning; and 
(3) Can be conducted in the form of scientific 
research and/or artistic production. It was also 
pointed out that though idiosyncrasy exists in 
individual concept structures, sufficient 
commonality and isomorphism in individual 
meanings make it possible to have dialogue and 
sharing. Therefore, being able to communicate 
and share concept structures within one's 
cognitive structure is the key to understand and 
evaluate meaningful learning. 

To better understand and assess meaningful 
learning, we need an effective tool to visualize it 
and Concept Map (CM) is such a tool. CM was 
introduced by Novak (Novak & Gowin, 1984) as a 
graphical tool for representing knowledge 
structure in the form of a graph. The nodes of the 
graph represent concepts. The edges that run 
between concepts represent relationships. 
Concepts and relationships between them 
formulate propositions. The simplicity of 
constructing a CM makes it an easy tool for 
anyone to represent her knowledge structure for 
others to see and understand (Canas et al., 
2005). Compared to other mapping techniques, 
CMs have solid underlying theories (Novak & 
Canas, 2008). 

To construct high quality CMs, one needs to 
constantly integrate newly acquired concepts and 
relationships into existing CMs, and the structures 
of the CMs need to be modified to accommodate 
the changes. The continuous iterative process of 
such integration signifies meaningful learning 
rather than rote learning. This makes CMs an 
excellent tool to visualize meaningful learning. In 
turn, the quality of CMs may be used to assess 
the magnitude and nature of meaningful learning. 

Figure 1 summarizes the relationship between 
CM, active learning, and assessment. The 
cognitive structure is a voluminous collection of 
concepts and their relationships. Meaningful 
learning is the iterative refinement and 
enrichment of this structure. The cognitive 


structure exists in one's mental world and is not 
directly accessible by others. Like a cognitive 
structure, CM is a graph collection of concepts and 
their relationships and can be iteratively refined 
and enriched. Unlike a cognitive structure, CMs 
exist in the physical world and can easily be 
accessed by others. In active learning using CM, 
a student captures new information in a CM and 
iteratively refines it (LI in Figure 1). This process 
in turn helps refine the cognitive structure, i.e. 
active learning (L2). In assessment, relevant 
portion of the cognitive structure is captured by a 
CM (Al), which can then be assessed (A2). 


Cognitive Structure; Active Learning; Iterative 

voluminous concepts modification of the 

and their relationships cognitive structure New Information 



Mental World Physical World 

Figure 1. Relationship between CM, Active 
Learning, and Assessment 


In this study, we focus on building various CM- 
based tasks into teaching in the IS curriculum at 
the University of Houston-Clear Lake (UHCL). 
Furthermore, the quality of completed CMs are 
analyzed both qualitatively and quantitatively. 
The analysis results provide us valuable insights 
on how students learn meaningfully. The rest of 
the paper is organized as follows. Section 2 
provides a survey on related theoretical and 
empirical work. Section 3 describes in detail the 
designed CM-based tasks, and their analysis and 
assessment. We then discuss the results in 
Section 4 and conclude with future research 
directions in Section 5. 

2. RELATED WORK 

The constructs used in CMs are simple and 
impose little cognitive burden on users-Concepts, 
Relationships, and Propositions. A concept is 
usually a word or a short phrase representing 
perceived regularity or pattern in events or 
objects, or records of events or objects. Generally 
speaking, there are two equally important 
categories of concepts in IS (Zendler, Spannagel, 
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& Klaudt, 2011). The first are content concepts 
such as algorithm, architecture, and data. The 
other are process concepts such as problem 
solving, problem posing, analyzing, and 
generalizing. The practical components focus on 
content concepts and corresponds to the 

technical-oriented classes in IS curricula such as 
DBMS. The theoretical components focus on the 
process concepts and corresponds to the 

theoretical-oriented classes in IS curricula such 
as IS Theory. Related concepts can be linked 
through relationships to formulate meaningful 
statements that represent the content and 
structure of one's knowledge body. A set of inter¬ 
connected CM constructs often suggest certain 
knowledge domain/field. Cross-domain links may 
occur if one's knowledge is comprehensive and 
the learning is meaningful since rote learning 
often remains at the "know-what" level. A simple 
concept map to explain what is concept map and 
how it is related to CT and meaningful learning is 
in Figure 2. 



Figure 2. Concept Map of Concept Map and 
Meaningful Learning. Partially adopted from 
(Canas et al., 2004) 

The underlying theory of CMs is cognitive learning 
(Ausubel, 1963, 2012) which builds on several 
principles. The key principle is meaningful 
learning. To facilitate meaningful learning, the 
learner must assimilate new knowledge (clear 
and relevant concepts and propositions) into 
existing cognitive structure. CMs is the perfect 
candidate for this task because the construction 
of a CM instantiates the process of conducting 
meaningful learning. Once the CMs are 
completed, we can gauge students' meaningful 
learning through the quality of the CMs. 
Therefore, we need to have effective 
methodology to evaluate the "goodness" of CMs. 


The criteria used in the evaluation of CMs usually 
measure the content and/or the structure of the 
CMs. The content evaluation of the CMs may 
measure various characteristics of CM 
components such as concepts, propositions, and 
their formed structures. The structure evaluation 
of the CMs usually looks at the 
interconnectedness of the CMs (Strautmane, 
2012; Yin, Vanides, Ruiz-Primo, Ayala, & 
Shavelson, 2005). Content evaluation often is 
based on a "master map"—a CM compiled to be 
used as the "gold standard". Structure evaluation 
often measures various topological characteristics 
of the CM. Flowever, there is no fixed formula of 
"goodness of CM" (Canas, Novak, & Reiska, 2015) 
since the "goodness" can be very subjectively 
based on various factors. For example, the 
purpose of CMs has an impact on what are to be 
considered as good CMs. The purposes may 
include knowledge elicitation, cognitive structure 
formation, assessment, etc. 

In addition, there are many different ways CM- 
based tasks can be designed and executed to 
represent knowledge and/or to assess learning, 
as summarized in (Strautmane, 2012). The 
variables of the tasks may include the following: 
(1) Whether a focus question is used 
(Derbentseva, Safayeni, & Canas, 2007)? A focus 
question provides a focal point for the learner to 
acquire, structure and assimilate a topic of 
knowledge. The CMs constructed accordingly 
should contain relevant concepts and their 
connections meaningfully organized to answer 
the focus question; (2) Whether certain types of 
assistance are provided by the instructors? For 
example, will part of the concepts, or structure, 
or both be provided to the constructor? Flow CM- 
based tasks are administered affects how CMs are 
constructed, and the quality of them in turn. 

As much as CMs are widely adopted in other 
disciplines, their application in IS education is 
rather limited. For example, in (Weideman & 
Kritzinger, 2003), thirteen applications of CMs in 
education are summarized, none of which is in a 
domain related to computing. In the limited cases 
where CMs are used in IS curriculum, assessment 
of the learning and knowledge structure is not the 
focus. For instance, CMs were adopted to gauge 
undergraduate students' understanding of 
content from MIS modules delivered in classroom 
setting (Gregoriades, Pampaka, & Michail, 2009) 
in order to test whether significant differences 
exist between Asian and European students 
learning styles and outcomes. Though CMs have 
been used to assess students' understanding, the 
scope is narrowed on a limited number of IS 
concepts (Freeman & Urbaczewski, 2001). In 
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other studies, CMs have also be used as a tool to 
teach and evaluate critical thinking in IS 
curriculum (Wei & Yue, 2016). 

The IS education community has a wide range of 
assessment tools, many of which have been 
proven effective in certain aspects, to some 
degree. Standard test questions such as multiple 
choice and T/F may be good at assessing "know- 
what"—usually results of rote learning. On the 
contrary, meaningful learning addresses "know- 
why" and "know-how". Writing assignments, 
hands-on projects, and case studies are often 
utilized for those. However, the deliverables of 
these assignments cannot effectively represent 
the cognitive processes and structures, which are 
important to understand the meaningful learning 
involved. The graphical structure that CMs 
provide can fit in this void. 

In this study, we take a holistic approach to 
integrate CM-based tasks as pedagogical tools 
into IS curriculum at UHCL. Different types of CM- 
based tasks are designed and executed. 
Mechanisms to evaluate the quality of the CMs are 
implemented. Tools are built to increase the 
automation level of the evaluation process. The 
evaluation results are interpreted based on 
theoretical and empirical work. This project is 
considered as the early phase of an effort to 
design and build a CM-Centered learning 
environment tailored to IS education (Canas & 
Novak, 2014). 

3. EXPERIMENT DESIGN 

In this study, we used five classes in three 
Computer Information Systems (CIS) courses at 
both graduate (G) and undergraduate (U) levels 
for testbed. Two major categories of IS courses 
are used: one type is technical oriented database 
classes where the focus is "content concepts" 
including definition, algorithm, data structure and 
more. The other is more theoretical oriented IS 
classes where the focus is "process concepts" 
including theories, frameworks, and problem 
solving procedures. The details of participating 
classes are summarized in Table 1. 

Our research focus is to explore "How CMs can be 
effectively used to assess meaningful learning in 
IS education?" More specifically, we would like to 
seek answers to the following questions: 

• What impact does CM-assignment design 
have on the outcomes? 

• How do students perform on CM-assignments 
and what are the insights? 


• Are there significant differences between CM- 
assignments performance of students at 
different academic levels? 

• Are there significant differences between CM- 
assignments performance of students from 
classes of different natures? 

• What features of CMs can be used to assess 
meaningful learning? More specifically, we 
would focus on the content and the structure 
of the CMs. 

• What modifications need to be made for 
future CM-assignments? 


Class 

# 

Course 

Level 

Concept 

Type 

1 

Design of Databases 
(DOD) 

U 

Content 

2 

3 

Infor. Systems Theory 
& Practice (ISTP) 

u 

Process 

4 

5 

Strategic Information 
Systems (SIS) 

G 

Process 


Table 1. Summary of Participating Classes 

CM-based Tasks 

For all participating classes, instructors prepared 
the students for the CM-assignments as follows: 
(1) Conduct brief in-class introduction of CMs with 
examples (around 20 minutes); (2) Distribute 
more learning material on constructing CMs for 
further reading; (3) Distribute CmapTools 
tutorials to help students grasp the diagramming 
tool they are going to use to complete the 
assignments; (4) Assign small in-class CM 
exercises and provide instructor feedback. Pre- 
CM short surveys were also conducted and the 
results show that the majority of the students had 
not been exposed to CM before. Afterward, the 
CM-assignments are distributed as regular 
homework assignments and students were given 
one week to complete them. 

For the purpose of constructing CMs, we adopted 
CmapTools (Canas et al., 2004). This tool was 
chosen over other diagramming tools because: 
(1) It is developed by the Florida Institute for 
Human and Machine Cognition (IHMC) based on 
their years' research on knowledge 
representation; (2) It is free for download and use 
for educational purposes; (3) It has an excellent 
user interface; (4) It provides network-based 
sharing and collaboration environment, which 
makes larger scale and longitudinal study on CMs 
possible; (5) It provides support to incorporating 
multimedia elements into the CMs; (6) It allows 
the CMs to be exported in various formats such 
as XML files, which makes it possible to automate 
some analysis of the CMs. 
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CM-construction assignments can come in 
different forms. For example, a focus question 
may be given to the students. Alternatively, an 
initial set of concepts may be provided to help the 
students to start on the construction. The given 
concepts can either be provided in a list or in a 
pre-defined structure. In this study, the details of 
the CM-assignments design for each participating 
class is summarized in Table 2. The focus 
question given to the ISTP class is "How could 
businesses develop competitive strategies using 
information systems?" For other classes, the CM- 
assignments are given based on specific teaching 
segments including "relational database model" 
(for one of the DOD classes), "Information 
Technology Architecture and Infrastructure (for 
SIS)", and "Social and Ethical Issues of 
information systems (for ISTP)". For the last one, 
the initial set of concepts provided to students 
include: Ethics, Accountability, Information 

Systems, Information, Moral dimension, Quality 
of life, Data, Piracy, Ethical issues, Intellectual 
property, Privacy, Control, Social issues, Political 
issues, Data analytics, Ethical analysis, Law, 
Security, Fair information practices, Ethical 
principles, Customer data, Computer crime. With 
this initial set, students are asked to construct a 
CM with at least 40 concepts. 


Class 

Focus 

Initial 

Sample 

# 

Question? 

Concepts? 

Size 

1 

N 

N 

28 

2 

N 

Y 

24 

3 

Y 

N 

26 

4 

N 

Y 

27 

5 

N 

Y 

19 


through them manually. Various studies have 
tried to use different techniques to analyze CMs, 
most of which have the focus of gauging the 
quality of the CMs (Canas, Bunch, Novak, & 
Reiska, 2013; Jain, Gurupur, & Faulkenberry, 
2013). Some other tools have the capabilities of 
comparing CMs to master CMs by seeking 
similarities (Lamas, Boeres, Cury, Menezes, & 
Carlesso, 2008; Marshall, Chen, & Madhusudan, 
2006). For our study, we designed and 
implemented Concept Map Analysis Framework 
(CMAF), a tool to analyze students' CMs. The 
design goals include: (1) Provide automated 
analysis and feedback to students who turn in 
CMs as assignment deliverables; (2) Provide 
summary reports of submitted CMs of a class to 
the instructor; (3) For each CM, provide a quality 
analysis report; (4) Provide results of comparison 
between student CM and the master CM. The 
framework is also designed in an extensible way 
so future research and teaching needs can be 
fulfilled. The architecture of CMAF is shown in 
Figure 3. 


Course 

Based 


{ 



Table 2 CM-Assignments Details 

Analysis and Evaluation of CMs 

The completed CMs are turned in electronically in 
both .cmap and ,cxl files. The .cmap file is the 
native file format for CMapTools and the ,cxl file 
is basically exported XML file that can be parsed 
to extract details of the CMs. The ,cxl files contain 
three major types of information: (1) General 
information of the CMs such as title, publisher, 
and date; (2) Content of the CMs including 
concepts (nodes), relationships (edges), and the 
labels of the nodes and edges; (3) Display 
information of the CMs such as the location of the 
nodes and edges, basically the graph layout 
information of the CMs. The first two types of 
information are useful in capturing and 
understanding the knowledge represented by the 
CMs and will be the foci of our analysis. 

Completed CMs have a lot of information 
embedded in them and it is impractical to go 


Figure 3. The Architecture of CMAF 

The tool is database-centric and implemented in 
Python. Students turn in their CMs labeled with 
their IDs. The CM Extractor extracts required 
elements from the CMs and stores them in the 
database (MySQL). Other relevant data such as 
course, assignment, and student information can 
also be used by the CM Extractor and the CM 
Database. CM Analyzer can retrieve CMs from the 
CM Database and the analysis results can be 
stored back to the CM Database. Report 
generators can generate appropriate reports 
upon request for different purposes. 

At this stage, the tool is capable of reading ,cxl 
files, parse and analyze the CMs, store the 
parsing and analysis results into a database, and 
generate various reports on CMs upon requests. 
The analysis of the CMs focuses both on the 
content and the structure of the CMs. Python 
NetworkX Package ("NetworkX-High Productivity 
Software for Complex Networks," 2014) is used 
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to deliver topological measures of the CMs. In the 
next phase, we plan to extend the tool's 
functionality by including similarity analysis, i.e., 
comparison between students' CMs and master 
CMs provided by the instructor. 

With the help of the tool, we were able to batch 
process the CMs. In addition to extraction and 
storing all components of the CMs, we also 
process the information to obtain a set of 
significant measures of the CMs. A summary of 
those measures is provided in Table 3 and Table 
4. Note that many of the structure measures are 
borrowed from standard Social Network Analysis 
(SNA) (Wasserman & Faust, 1994). 


Measure 

Definition 

n_nodes 

Number of concepts in CM 

n_edges 

Number of linkages between 
pair of concepts in CM 

n_chars 

Number of characters in the 
labels 

n_words 

Number of words in the label 


Table 3 Captured Content Measures of CMs 


Measure 

Definition 

n_center 

Number of nodes that are 
centers 

n_periphery 

Number of nodes that are 
periphery nodes 

density 

Graph density 

is_connected 

Boolean value to denote if 
the CM is connected or not 

radius 

Minimum eccentricity 

diameter 

Maximum eccentricity 

deqree 

Number of edqes for a node 

in_deqree 

Number of incoming edges 

out_deqree 

Number of outqoinq edges 

deg_cent 

Degree centrality 

close_cent 

Closeness centrality 

between_cent 

Betweenness centrality 


Table 4 Captured Structure Measures of CMs 


As an example, Appendix 1 shows a CM created 
by an above-average student in the 
undergraduate DoD class in a CM assignment to 
capture concepts in the relational databases and 
the relation model by using CMAP. Table 5 shows 
the values of captured content and structured 
measures of the CM. 

This information is useful in assessment and 
providing feedback to the student. Appendix 2 


shows a feedback report generated by CMAF to 
the student producing the CM in Appendix 1. 

CMAF is currently under active development and 
we will present it in more details in a future paper. 
Meanwhile, readers interested in learning more 
about CMAF may contact the authors. 


Measure 

Sample CM Value 

n_nodes 

28 

n_edges 

37 

n_chars 

12.43 

averaqe(n_words) 

1.82 

averaqe(n_center) 

3 

n_periphery 

6 

density 

0.098 

is_connected 

true 

radius 

4 

diameter 

7 

averaqe(deqree) 

0.98 

averaqe(in_deqree) 

0.049 

average(out_deqree) 

0.049 

averaqe(deg_cent) 

0.3 

averaqe(close_cent) 

0.095 

averaqe(between_cent) 

0.095 


Table 5 Graph Measures of Sample CM in 
Appendix 1 


4. ASSESSMENT RESULTS AND DISCUSSION 

Due to the limited space, we select only part of 
our analysis results for description and discussion 
in this paper as follows. 

Grading CMs against Master CM 

One way to evaluate the quality of a student's CM 
is to compare it against the master CM provided 
by the instructor. This process can be very time 
consuming since automation of this process is 
hard to achieve. Because of the free form of 
concepts, relationships, and propositions, 
detailed grading of CM elements requires manual 
work and domain expertise. 

Scoring of CM based on quality of the elements 
have been studied (McClure & Bell, 1990; 
McClure, Sonak, & Suen, 1999). We adopted and 
modified the previous scoring methods to 
evaluate students' work. Basically, the instructor 
created a "master CM", against which student 
work were compared to obtain Hoiistic Score, 
Existential Score, and Relational Score. Holistic 
score was used to assess the overall 
understanding of the content (i.e., the subject 
matter). The Holistic Score measures the "general 
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goodness" of the CMs and is often assigned by the 
graders who are familiar with the purpose of the 
assessment. Existential score captures the 
presence or lacking of required concepts, 
weighted by their relative significance in the CM. 
CMs that contain more "significant" concepts in 
the master CM scores higher in this aspect. 
Relational score measures the existence and 
correctness of relationships between concepts, 
and relationships are also weighted. CMs that 
include more heavy-weighted relationships score 
higher in this aspect. These three different scores 
were combined in a weighted-manner to compute 
the overall score. The overall score is calculated 
on a 1-10 scale as Overall = (10 x —+ 10 x + 

Emax E max 

H)/ 3, where E and R are the Existential and 
Relational scores respectively. E ma x and R ma x are 
the highest achievable existential and relational 
scores and they can be calculated using the 
master CM. The graders, based on their 
understanding of the content, also assign the 
weights of the concepts and relationships. H is the 
holistic score on a 1-10 scale and the assignment 
of a value for H relies on the grader's criteria and 
domain knowledge. Using this method, completed 
CMs by students were graded and the general 
findings are as follows: (1) Students tend to 
achieve higher existential score than relational 
score; (2) Overall high score is rare compared to 
the master CM; (3) High holistic score doesn't 
necessarily correlate with high existential and/or 
relational scores; (4) Grading score, especially 
the relational score, correlates positively with 
course grade. A possible implication of this is that 
students who are better in meaningful learning 
(required to achieve high relational scores) 
generally perform better than others in the class, 
where knowing and memorizing facts is not 
sufficient. In addition, by observing the CMs by 
students, instructors can gain insights as to how 
to improve teaching to facilitate meaningful 
learning such as: (1) What concepts do many 
students fail to include in the CMs, especially 
those concepts that are essential to learning 
objectives? The instructor may consider modify 
teaching to emphasize those important concepts. 

(2) What are the commonly missed/incorrectly 
labeled relationships that need more clarification? 

(3) Is the teaching structured in the way to help 
students see connection between topics? This can 
be done by observing the existence and/or 
absence cross-topic relationships. Currently, 
instructors do most of the grading against master 
map manually. We plan to include at least part of 
this process into our CMAF. 


General Features of CMs 

Some general features of CMs include: (1) The 
number of concepts (nodes) in a CM (#N); (2) 
The number of relationships (edges) in a CM 
(#E); (3) Whether the CM is connected (C); and 

(4) Number of words (NW) in the edge labels of a 
CM. In Table 6, the mean and standard deviation 
of node count and edge count compared to those 
of master CMs are summarized. 


c 

#N 

#E i 

# 

Avg 

Std 

Mast. 

Avg 

Std 

Mast. 

1 

28.8 

19.6 

20 

29.1 

19.7 

24 

2 

25.1 

5.0 

30 

29.9 

6.5 

43 

3 

27.1 

11.8 

40 

36.8 

18.9 

47 

4 

46.9 

9.6 

55 

53.5 

12.1 

58 

5 

49.8 

19.1 

60 

54.4 

23.3 

65 


Table 6 CMs Nodes and Edges Count 

For technical classes (Class 1), average numbers 
of concepts and relationships from students' work 
are 43% and 22% more than those of the master 
CM. This assignment doesn't have a focus 
question or any initial concepts to start with, 
which leaves the solution space wide open. In- 
depth analysis of CMs from Class 1 suggests that 
the CMs (1) Are less connected; (2) Have higher 
number of distinct concepts and relationships; (3) 
Have more verbose concepts; and (4) Have less 
verbose relationship labels. 

For IS theory classes (Classes 4 and 5) with initial 
concepts provided, the average number of 
concepts and edges provided by the students are 
closer to those of the master CMs (85.5% of 
nodes and 93.1% of edges for Class 4, 83.3% of 
nodes and 83.1% of edges for Class 5). 
Therefore, the initial given concepts help improve 
the coverage of necessary concepts and set the 
proper scope of the concepts. In addition, it can 
be seen that standard deviation of edge count is 
usually significantly higher than that of node 
count, which suggests that students' capabilities 
in creating meaningful relationships between 
concepts vary more compared to their capabilities 
in coming up with concepts. Teaching tools should 
be designed to help students see connections 
between what they have learned. 

We view the complete CMs as graphs, a 
disconnected CM means there are segments not 
connected to others and each segment usually is 
a topic/subdomain. Disconnected CM suggests 
that the author has trouble establishing 
connections between topics in the same 
knowledge area. Obviously, the cross-topic 
connections should carry more value when 
measuring the quality of CM since "putting the 
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whole picture together" requires true learning in 
depth. Our analysis results give some insights on 
this matter as follows: (1) The two classes with 
focus on "content concepts" (database 
technologies) have much higher percentage of 
connected CMs, i.e., no broken pieces in the CMs 
(89.3% and 95.8% respectively). The three 
classes with focus on "process concepts" (IS 
theories) perform worse and the connected 
percentages are 56.0%, 44.4%, and 78.9%. For 
the knowledge area of DBMS, the content and 
structure are more maturely established and 
stable, which makes it easier for the students to 
see the holistic view. For IS theory classes, the 
topics are more diverse and students tend to lose 
track of the connectedness. Flowever, with 
advancement in the program, this aspect gets 
improved as we can see graduate students 
(78.9%) perform much better than 
undergraduate students. Furthermore, we also 
found that in IS theory classes, CMs have higher 
number of words in the concept labels than DBMS 
classes. This often happens because concepts in 
IS theory classes are more abstract and students 
have more trouble in coming up with precise and 
succinct concepts. In some extreme cases, a 
whole sentence is used as a concept. What the 
students fail to realize is that very long concept 
label is a good indication that more complicated 
structure such as propositions should be used 
instead, as seen in the example shown in Figure 
4. 


[ Ethics ]-is- 


The principles of right and wrong 
that inividuals, 
acting as free agents, 
use to make choices 


[choices] 

ir 

make 

[ Ethics ]—is—►[ Principles]— 

\ 

[ People ]— 


Right & 
Wrong 


use_ 
•as— 


Free 

Agents 


Figure 4. Example of a Very Long Concept 

Structure Features of CMs 

In this section, we illustrate our findings by 
analyzing CMs as graphs using network analysis 
techniques provided in NetworkX, with focus on 
selected features. For a node in a graph, its 
eccentricity measures the longest distance 
between it and any other nodes. The minimum 
eccentricity of a graph is its radius and the 


maximum eccentricity is the diameter. The nodes 
whose eccentricity equals to the radius are called 
center. The nodes with eccentricity equals to the 
diameter are called periphery. For a node, the 
number of edges connected to it is called the 
degree. For directed graph, there are in-degree 
and out-degree. Centrality is used to measure the 
relative importance of a node in a graph, based 
on how connected is this node to others. Four 
different centrality measures are studied 
including degree, betweenness, closeness, and 
load centrality (Wasserman & Faust, 1994). 


Average Radius and Diameter of CMs 


"03 **■ 



Class 1 Class 2 Class 3 Class 4 Dass 5 


■Avg.Radius "Avg. Diameter 

Figure 5. Comparison of Radius and Diameter 

As seen in Figure 5, CMs from DOD classes 
(Classes 1 & 2) are more "round" and CMs from 
the IS theory classes have more "spikes" because 
the diameters are longer. In other words, we tend 
to see longer chains of concepts in IS theory CMs. 
It indicates those CMs are more of depth and 
suggests hierarchies. Going through the details of 
the CMs, it is discovered that some most popular 
relationships between concepts are "is a", "is type 
of", and "is part of" and their variations. In the 
completed CMs, the largest value of diameter is 
15 (in the undergraduate IS theory class), which 
means the author was able to expand from one 
concept to another as far as 15 steps. 

Degree of a node measures how many other 
nodes it connects to. In the case of CM, for each 
concept, its degree indicates how many other 
concepts are connected to it. For all collected 
CMs, we calculate their average degrees, i.e., 
generally each concept in the CM is linked to how 
many other concepts. This measure and its range 
vary significantly cross the classes, as seen in 
Figure 6. The graduate IS theory class has the 
widest range of average degree count compared 
to others. 

In addition, we conducted t-tests to find out if 
significant differences exist between the means of 
average degree counts. The results are 
summarized as follows. 
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• Between the two databases classes, the class 

that was given an initial set of concepts to 
start with has significantly higher average 
degree count (t=-5.1392, df=42.536, 

p<0.0001). 

• Between the two undergraduate IS theory 
classes, the class that was given a focus 
question to start with has significantly higher 
average degree count (t=-2.3047, df = 
35.971, p=0.01). The highest average degree 
count is 15 and it happens in one of the CMs 
where the concept "Information Systems" is 
the center of the CM and has links to many 
other lower level topics. 


topics organized in the CMs. Therefore, IS theory 
CMs should have higher betweenness centrality. 
Using our collected CMs data, we performed t 
tests to test our hypothesis and the conclusions 
are drawn as follows: (1) The database classes 
CMs have significantly higher degree centrality 
than IS theory classes (t = 3.4796, df = 120.242, 
p<0.001); (2) The IS theory classes CMs have 
significantly higher betweenness centrality than 
database classes (t = -6.5823, df = 192.602, p < 
0.0001). These findings provide us insights how 
to design CM assignments to encourage higher 
quality work based on different nature of the 
knowledge areas in IS. 


These observations inform us that by providing an 
initial set of concepts and/or a focus question, we 
can encourage students to seek more 
relationships between concepts. Probably the 
starting concepts and focus question can act as 
anchors of the CMs. 


Average Degree of Cone 

6 

C _' 

epts in Students' CMs 

J 

A 


4 

o T 

. T 

, _L i 

1 1 * 

i 

Classl Class2 Class3 Class4 Class5 


Figure 6. Boxplots of Average Degree of 
Concepts in CMs for All Five Classes 

In SNA, centrality is a measure to represent the 
significance of a node. There are different types 
of centrality measures. Degree centrality is 
defined based on the degree of a node, i.e., the 
number of edges between the node and its 
neighbors. In CMs, a node with high degree 
centrality signifies important concepts, i.e., 
central ideas in the knowledge area. Between 
centrality quantifies the number of times a node 
acts as a bridge along the shortest path between 
two other nodes. In CMs, a node with high 
betweenness centrality is a concept that act as 
gateway between topics within the domain. A CM 
contains high betweenness centrality concepts 
suggests that the author has a holistic view of the 
learning content. The central concepts from the 
database classes are more well-defined and the 
CMs should have higher degree centrality. As to 
the IS theory classes, contents covered are more 
dispersed and we expect to see many related 


5. CONCLUSIONS AND FUTURE RESEARCH 

CM is an effective tool to represent one's 
knowledge. The content and quality of CMs can 
provide valuable insights into what and how the 
authors have learned. In this study, we designed 
a series of CM-based assignments to understand 
students' meaningful learning in two IS courses- 
a technical and a theory class. We also designed 
and implemented a tool to extract elements from 
the students' CMs and conducted various analysis 
of the results. From our study, we gained the 
following insights: 

• CMs are an excellent tool from which 
instructors can gauge students' learning and 
improve teaching. 

• Learning curve to CMs and CmapTools is 
short, which makes incorporation of it into the 
teaching feasible. 

• CM-based assignments come in different 
formats and this has an impact on the 
outcomes including whether a focus question 
or initial concepts are provided. For example, 
proper focus questions and initial set of 
concepts can improve the quality of the 
students' CMs, especially for IS theory 
classes. 

• CMs constructed for different classes in IS 
curriculum vary in many features and those 
should be taken into consideration when 
designing the assignments. 

• Quantitatively grading the CMs using master 
CMs requires time and expertise. Though the 
grading can provide interesting findings, one 
should be cautious against using the scores 
without proper interpretation. 

We believe there is a lot more to be explored 
about the usefulness and utility of CMs in IS 
education, especially to understand students' 
learning. Our current works can be considered as 
pilot studies on a graphical tool with high 
potential in IS education. Our experimental 
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designs are limited by the small sample sizes, the 
small number and variety of participating IS 
classes, the absences of control groups, and the 
lack of a strong theoretical model. Furthermore, 
we have tested only a few variety of CM 
assignments. As a flexible graphical tool, the kind 
of CM assignments can be very rich and a 
taxonomy of these CM assignments in the context 
of IS education has not been studied 
systematically. Both the assessment methods 
and the CMAF tool are in their early stages and 
much can be improved. Based on the lessons 
learnt in this series of preliminary studies, we will 
address these limitations and expand the scope 
and depth of our study and continue to improve 
our CMAF. 
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Appendix 1 An example CM of a student taking the undergraduate database class 
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Appendix 2. CMAP report to the student creating the CM in Appendix 1 


CMap Report 


HW #3 Concept Map 
CSCI 4333 spring 2016 

Number of students: 24 
Average number of concepts: 25.12. 
Average number of links: 30.33. 
Average connectivity: 1.21. 

Suggested model solution: 

Number of concepts: 30. 

Number of links: 43. 
Connectivity: 1.43. 

Student id: xxxxxxx 


Number of concepts: 28. 

Number of links: 37. 

Connectivity: 1.32. 

Concepts and number of edges coming in and out from them, 
n Concept # from # to #total 


1 relation (or table) 

2 SQL Queries 

3 Relational DBMS 

4 fields 

5 primary key 

6 operations 

7 super key 

8 integrity 

9 tuple (or row) 

10 Relational database 

11 relation instance 

12 composite key 

13 relation schema 

14 column value 

15 foreign key 

16 columns 

17 rows 

18 referential integrity 

19 candidate key 

20 column type 

21 integrity constraints 

22 secondary key 

23 degree 

24 RDB engine 

25 extension 

26 alternate key 

27 Relational Query Language 

28 DBMS 


3 5 8 

5 1 6 

2 2 4 

0 4 4 

2 2 4 

3 1 4 

3 1 4 

12 3 

2 1 3 

2 1 3 

3 0 3 

12 3 

2 1 3 

11 2 

0 2 2 

0 2 2 

0 2 2 

2 0 2 

11 2 

0 2 2 

10 1 

10 1 

10 1 

10 1 

0 1 1 

0 1 1 

0 1 1 

0 1 1 
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